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Abstract 


In  this  report,  three  prototype  displays  that  are  contrasted  for  aircraft  navigation  and  tactical 
hazard  awareness:  a  conventional  2D  coplanar  display,  an  exocentric  (“God's  eye”)  3D  display,  and 
an  immersed  (“pilot's  eye”)  3D  display.  Strengths  and  weaknesses  of  each  display  type  are 
discussed.  In  implementing  these  for  Experiment  1,  the  immersed  display  was  coupled  with  a 
global  exocentric  view  of  the  airspace.  In  Experiment  1,  thirty  pilots  flew  with  each  of  the  displays 
around  an  airspace  populated  by  waypoints,  hazards  and  traffic.  Performance  measures  revealed  the 
anticipated  costs  and  benefits  of  each  display  type.  In  Experiment  2,  cognitive  engineering 
principles  were  applied  to  remediate  the  weaknesses  of  each  display  type.  Analysis  of  performance 
of  27  additional  participants  in  flight  path  tracking  and  hazard  awareness  measures  revealed  success 
in  application  of  the  cognitive  engineering  principles.  We  also  briefly  describe  the  procedures 
employed  in  a  third  experiment  in  which  additional  design  changes  were  examined. 
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Introduction 


Rapid  advances  in  computer  graphics  technology  have  made  feasible  a  variety  of  possible 
formats  of  dynamic  three-dimensional  graphics  for  aviation  displays.  Options  may  include 
panoramic  displays  (Busquets,  Parrish,  &  Williams,  1992),  “highway  in  the  sky”  displays  (Reising, 
Barthelemy,  &  Hartsock,  1989;  Haskell  &  Wickens,  1993;  Barfield,  Rosenberg,  &  Furness,  1995; 
Flohr  &  Huisman,  1997;  Theunissen,  1995),  “virtual  cockpits”  (Haas,  1992)  or  somewhat  more 
conventional  3D  electronic  maps  (Wickens,  Liang,  Prevett,  &  Olmos,  1996;  Olmos,  Liang,  & 
Wickens,  1997;  Wickens  &  Prevett,  1995).  On  the  one  hand,  the  concept  of  three-dimensional 
displays  makes  intuitive  sense,  because  the  pilot's  world  is  a  very  three-dimensional  one.  Yet 
empirically,  whether  the  benefits  of  3D  displays  outweigh  their  costs  turns  out  to  be  a  complex 
issue,  depending  upon  the  particular  3D  rendering  chosen,  the  nature  of  the  task,  and  the  structure 
of  the  information  to  be  displayed.  We  consider  these  issues  in  turn. 

Display  viewpoints.  We  first  note  that  there  are  two  qualitatively  different  viewpoints 
offered  by  3D  displays,  shown  to  the  left  of  Figure  1.  The  “immersed”  or  fully  egocentric  viewpoint 
(panel  A)  presents  the  world  much  as  a  pilot's  eyes  might  see  it  from  the  cockpit  (Theunissen,  1995; 
Flohr  &  Huisman,  1997;  Jensen,  1981).  The  exocentric  viewpoint  (panel  B)  presents  a  view  from 
behind  and  above  ownship.  In  many  respects,  these  two  3D  viewpoints  are  as  different  from  each 
other,  as  both  are  from  the  more  conventional  2D  display  (panel  C),  with  which  they  are  often 
compared.  However,  we  note  that  any  fair  comparison  of  3D  with  2D  viewpoints  must  provide  the 
latter  with  two  orthogonal  views  of  the  airspace  so  that  all  viewpoints  are  provided  with  a  spatial 
analog  representation  of  the  vertical  axis  (Fadden  ,  Braune,  &  Wiedemann,  1991).  Hence,  we  refer 
to  this  as  the  coplanar  view. 

Tasks.  One  may  distinguish  at  least  four  categories  of  spatially  relevant  aviation  tasks  which 
the  above  viewpoints  must  serve.  Local  guidance  or  navigation  simply  requires  adherence  to  a 
flightpath  leading  forward  from  the  nose  of  the  aircraft.  Navigational  checking  in  visual 
meteorological  conditions  requires  cross  referencing  map  information  with  visual  information  in 
the  forward  view,  to  ascertain  congruence  between  the  two  (Hickox  &  Wickens,  1996),  and  thereby 
establishing  that  the  pilot  is  geographically  oriented.  Spatial  or  hazard  awareness,  a  subset  of  the 
more  generic  concept  of  situation  awareness,  requires  knowledge  of  the  location,  orientation  and 
trend  of  entities  (weather,  other  traffic,  no-fly  zones,  terrain),  lying  within  a  much  broader  sphere  of 
the  surrounding  airspace.  Hazard  awareness  can  be  broken  down  in  terms  of  the  degree  of  precision 
with  which  location  and  trend  must  be  determined.  For  example,  a  traffic  display  for  collision 
avoidance  must  yield  very  precise  location  and  trend  information.  In  contrast,  a  combat  display  for 
general  strategic  planning,  or  a  weather  display  for  route  planning  imposes  less  precise 
requirements. 
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Figure  1.  Three  canonical  viewpoints  for  aviation  displays  and  the  resulting  display  depiction:  (a) 
egocentric  or  immersed;  (b)  exocentric  or  “tethered”;  (c)  coplanar.  (a)  and  (b)  are  considered  “3D” 
displays;  (c)  is  a  2D  display  under  which  each  display  is  listed  the  associated  human 
perceptual/cognitive  costs. 


Information.  The  information  itself,  upon  which  the  above  tasks  are  dependent,  may  vary  in 
its  distribution  across  a  3D  volume  in  ways  that  appear  to  influence  the  effectiveness  of  different 
display  viewpoints.  Such  information  may  lie  along  constrained  2D  planes  (e.g.,  such  as  a  flat 
segment  of  terrain,  or  the  flight  path  of  a  level  aircraft)  or  may  be  distributed  as  less  constrained  3D 
volumes  or  trajectories  (e.g.,  the  3D  volume  of  hazardous  weather  region,  or  the  3D  flight  path  of 
an  attacking  hostile  aircraft).  In  the  former  case,  the  dimensional  constraints  may  be  assumed  by  the 
user  perceiving  the  display,  imposing  some  level  of  top-down  perceptual  processing.  In  the  latter 
case,  these  constraints  cannot  be  imposed. 

Empirical  data.  Different  display  viewpoints  appear  to  be  differentially  effective  for  the 
different  tasks  discussed  above.  The  most  robust  finding  regarding  the  costs  and  benefits  of  the 
three  prototype  display  viewpoints  shown  in  Figure  1  is  that  the  immersed  “tunnel  in  the  sky” 
display  is  superior  to  the  other  two  prototypes  for  guidance  tasks  (Barfield  et  al.,  1995;  Haskell  & 
Wickens,  1993;  Wickens  &  Prevett,  1995;  McCormick  &  Wickens,  1995),  and  for  navigational 
checking  (Hickox  &  Wickens,  1996).  The  advantage  for  guidance  appears  to  result  both  from  the 
compatibility  or  congruence  between  the  location  of  the  display  viewpoint  and  the  axis  of  control 
rotation,  and  because  of  the  higher  gain  by  which  rotational  motion  is  represented  on  the  display. 
However,  the  same  immersed  viewpoint  appears  to  be  inferior  as  a  means  of  providing  hazard 
awareness  information.  This  inferiority  results  either  because  the  display  provides  only  a  “keyhole” 
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view  of  the  world  behind  and  around  ownship  (Woods,  1984),  or  because  an  expanded  geometric 
field  of  view  (designed  to  address  the  keyhole  phenomenon)  severely  distorts  the  perceived  position 
of  hazards  (Wickens  &  Prevett,  1995). 

In  contrast  to  the  clearly  demarked  findings  for  the  immersed  display,  the  differences 
between  coplanar  and  exocentric  formats  are  more  subtle,  as  they  depend  upon  a  tradeoff  between 
two  information  processing  mechanisms.  On  the  one  hand,  the  coplanar  format  imposes  on  the  pilot 
a  visual  scanning  requirement  in  order  to  fixate  the  two  views,  and  an  added  cognitive  load  required 
to  mentally  integrate  them.  On  the  other  hand,  the  exocentric  format  suffers  from  the  ambiguity 
with  which  the  3D  location  of  entities  is  collapsed  onto  the  2D  viewing  surface  (Gregory,  1973; 
McGreevy,  &  Ellis,  1986).  Location  of  either  ownship  or  hazards  is  imprecisely  determined  along 
the  line  of  sight,  and  the  perceived  orientation  and  length  of  the  vector  connecting  the  two  (i.e.,  the 
azimuth,  elevation  angle,  and  distance  from  ownship  to  a  hazard)  will  be  degraded  by  the  imprecise 
estimates  of  both  endpoints  of  the  vector  (McGreevy  &  Ellis,  1986). 

Our  research  findings  have  indicated  that  the  contributions  of  either  scanning  or  ambiguity 
to  the  costs  and  benefits  of  coplanar  versus  exocentric  displays  will  be  modulated  by  a  number  of 
other  factors.  For  example,  to  the  extent  that  the  combined  implications  of  change  on  all  three 
dimensions  is  relevant,  the  scanning  costs  of  the  coplanar  display  appear  to  impose  a  greater  penalty 
(Wickens,  Merwin,  &  Lin,  1994;  McCormick  &  Wickens,  1995).  But  to  the  extent  that  there  is  a 
need  for  precise  estimation  of  3D  position  (e.g.,  estimating  the  position  and  trajectory  of  air  traffic), 
then  the  3D  ambiguity  costs  begin  to  dominate,  as  revealed  by  our  findings  in  air  traffic  control 
(May,  Campbell,  &  Wickens,  1996;  Wickens,  Miller,  &  Tham,  1996),  and  cockpit  traffic  displays 
(Merwin,  O’Brien,  &  Wickens,  1997).  However,  even  this  3D  ambiguity  cost  for  precise  location 
estimation  appears  to  be  mitigated  by  the  extent  to  which  the  viewer  assumes  that  properties  within 
the  environment  constrain  the  ambiguity  of  positions  (e.g.,  knowledge  that  all  points  lie  along  a 
level  surface  or  flight  path).  This  explains  why  3D  ambiguity  problems  in  trajectory  estimation  are 
not  observed  with  level  traffic  in  either  air  traffic  control  displays  (Wickens,  Miller,  &  Tham,  1996) 
or  cockpit  traffic  displays  (Merwin  et  al.,  1997). 

Finally,  in  circumstances  in  which  there  is  not  a  requirement  for  high  precision  of  spatial 
estimates,  and  in  which  some  constraints  on  3D  spatial  behavior  are  present  (e.g.,  the  continuous 
and  predictable  descent  of  an  aircraft  on  approach  to  landing),  it  is  not  surprising  that  our  results 
have  revealed  rough  equivalence  between  the  two  display  types  (Wickens,  Liang,  Prevett,  &  Olmos, 
1996;  Wickens  &  Prevett,  1995),  and  have  sometimes  revealed  a  trend  in  favor  of  the  3D  display, 
with  its  reduced  scanning  requirements  (Olmos  et  al.,  1997). 

Many  of  the  empirical  comparisons  between  2D  and  3D  displays,  to  support  both  guidance 
and  hazard  awareness,  have  been  carried  out  in  the  context  of  civil  aviation,  in  which  maneuvering 
performance  is  relatively  constrained  by  “rules  of  the  airspace.”  However,  in  many  aspects  of 
combat  tactical  maneuvering,  such  constraints  are  entirely  absent,  suggesting  two  competing 
alternatives.  On  the  one  hand,  it  is  possible  that  a  high  degree  of  3-dimensional  maneuvering  might 
impose  a  greater  cost  of  scanning  and  cognitive  integration  with  the  coplanar  display,  rendering  it 
inferior.  Such  an  interpretation  can  be  offered  to  explain  the  superiority  of  3D  over  coplanar 
displays  in  data  visualization  experiments  carried  out  by  Wickens  et  al.  (1994)  and  by  McCormick 
and  Wickens  (1995).  On  the  other  hand,  it  may  be  that  when  the  constraints  on  information  location 
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and  trajectories  are  removed,  this  will  amplify  the  ambiguity  costs  associated  with  3D  (perceptive) 
displays  in  hazard  location  awareness. 

In  order  to  investigate  these  issues,  two  experiments  are  reported  here  that  evaluated  the 
relative  merits  of  the  three  display  prototypes  in  supporting  3D  flight  path  tracking,  and  tactical 
hazard  awareness  regarding  the  3D  volume  of  airspace  around  a  simulated  combat  aircraft.  In 
Experiment  1,  we  compare  performance  of  the  three  “baseline”  displays.  However,  we  make  one 
important  and  necessary  modification  to  the  egocentric  viewpoint  (Figure  la).  Since  the  keyhole 
effect  associated  with  this  viewpoint  is  unacceptable  for  hazard  awareness  in  air-to-air  combat 
(hostile  targets  in  the  rear  quarter  often  represent  the  greatest  hazards),  the  immersed  forward  view 
(a  “local  display”)  was  coupled  with  a  large  scale  (but  small  resolution)  global  display  positioned  in 
the  lower  portion  of  the  viewing  area.  Hence,  we  refer  to  this  as  the  “split  screen”  display.  Thus,  all 
three  displays  depict  the  same  region  of  airspace  on  the  same  display  screen  size,  but  from  different 
viewpoints  (or  combinations  of  viewpoints).  In  Experiment  1,  participants  flew  to  a  set  of 
waypoints  positioned  among  a  series  of  hazards,  and  were  required  periodically  to  orient  to  air 
targets.  We  anticipate  the  experimental  data  to  reflect  the  strengths  and  weaknesses  of  each  display 
type  for  the  set  of  tasks  employed,  as  discussed  above.  Then  in  Experiment  2,  we  attempt  to  apply 
cognitive  engineering  principles,  to  address  the  inherent  weaknesses  of  each  display  type,  without 
simultaneously  encumbering  them  with  alternative  shortcomings. 

Experiment  1:  Baseline  Comparison 

Participants.  Thirty  University  of  Illinois  Institute  of  Aviation  personnel,  all  licensed 
pilots,  participated  in  the  experiment,  each  receiving  payment  of  $5.00  per  hour.  Flight 
experience  varied  from  65  to  2000  hours  with  a  median  time  of  250  hours.  Participants  ranged 
from  18  to  45  years  of  age.  All  participants  flew  with  all  three  display  formats.  However,  before 
flying  with  each  unique  display  condition,  participants  were  given  a  short  practice  session  to 
familiarize  themselves  with  the  display  format.  All  participants  received  the  same  instructions 
and  performance  measures. 

Apparatus  and  Flight  Dynamics 

The  study  was  conducted  on  a  Silicon  Graphics  IRIS  workstation  with  a  16-inch  diagonal 
screen.  Navigation  was  controlled  with  a  two-degree-of-freedom  joystick  which  was  attached  to 
the  right  arm  of  the  participant's  chair.  Standard  aviation  dynamics  were  employed  so  that 
pushing  forward  on  the  joystick  caused  the  aircraft  to  pitch  down  with  a  subsequent  loss  of 
altitude,  while  pulling  back  on  the  joystick  caused  the  aircraft  to  pitch  up,  thereby  increasing 
aircraft  altitude.  The  degree  of  pitching  up  or  down  directly  controlled  the  rate  of  altitude  change. 
Moving  the  joystick  to  the  right  or  left  made  the  aircraft  roll  right  or  left,  respectively.  Roll  angle 
was  directly  proportional  to  the  rate  of  heading  change  and  the  aircraft  was  not  allowed  to  roll 
past  90  degrees  in  either  direction.  Pitch  and  roll  were  directly  coupled  such  that  for  a  given  roll 
rate  the  aircraft  would  have  a  proportional  tendency  to  pitch  down.  Power  was  held  constant, 
although  airspeed  was  allowed  to  vary  from  180  knots  at  level  flight  to  a  minimum  of  160  knots 
in  a  climb  and  a  maximum  of  190  knots  in  a  descent.  There  were  no  throttle  or  rudder  controls. 
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Display 

The  world  in  which  the  pilots  flew,  with  respect  to  the  location  of  both  terrain  and  hazard 
volumes,  was  the  same  across  all  display  conditions.  Terrain  was  represented  with  transparent 
geometric  wire  frame  objects  and  hazard  volumes  were  depicted  by  inverted  radar  coverage 
cones  (Figures  2,  3,  &  5).  To  reduce  any  carryover  effects  across  display  conditions,  each  session 
with  a  new  display  began  with  the  aircraft  in  a  different  location  and  at  a  different  altitude.  In 
addition  to  changing  aircraft  location,  the  location  of  the  various  waypoints  was  also  altered  for 
each  unique  display  condition.  A  rotating  map  was  employed  for  all  three  display  formats. 

2D  coplanar  display  (Figure  2):  For  the  2D  display,  both  a  planar  and  a  profile  view  were 
provided  to  present  lateral  and  vertical  information,  respectively.  The  planar  (horizontal 
situation)  view  was  placed  at  the  top  of  the  display  and  presented  information  from  a  bird’s  eye 
perspective;  hence,  no  vertical  information  was  provided  in  this  viewpoint.  To  ensure  depiction 
of  threats  in  front  of  and  behind  ownship  the  participant’s  aircraft  was  placed  at  the  center  of  the 
display.  The  profile  view  was  placed  at  the  bottom  of  the  display  and  presented  vertical 
information  from  a  viewpoint  orthogonal  to  the  aircraft's  moment-to-moment  location,  with  a  0° 
azimuth  angle  (parallel  to  the  fuselage),  and  at  an  elevation  angle  of  0°.  As  with  the  plan  view, 
the  pilot’s  aircraft  was  placed  at  the  center  of  the  profile  view  to  ensure  adequate  threat  depiction. 
To  aid  vertical  control,  a  predictor  in  the  form  of  two  poles,  one  at  the  front  and  one  at  the  rear  of 
the  aircraft  and  extending  to  a  grid  surface  below  the  aircraft  was  added  to  the  aircraft  icon. 

3D  exocentric  display  (Figure  3):  The  perspective  display  combined  lateral  and  vertical 
information  into  a  single  display  by  employing  a  “tethered”  exocentric  viewpoint  (Wickens  & 
Prevett,  1995).  The  display  viewpoint  was  placed  at  a  distance  of  40,000  display  feet  from  the 
aircraft.  A  vertical  predictor  similar  to  that  used  in  the  2D  condition  was  used  to  help  provide 
altitude  information.  In  order  to  ensure  that  the  predictor  did  not  lie  along  the  line  of  sight  of 
projection  and  hence  become  invisible,  an  azimuth  offset  of  15°  was  used.  A  geometric  field  of 
view  of  50°  was  employed  to  help  with  the  depiction  of  threats  beside  and  behind  ownship.  An 
elevation  angle  of  30°  was  also  used  to  assist  the  participants  with  vertical  judgments.  These 
projection  parameters  for  the  exocentric  viewpoint  are  schematically  depicted  in  Figure  4. 

3D  split-screen  display  (Figure  5):  The  split-screen  display  combined  two  differing 
perspective  viewpoints,  both  of  which  employed  a  50°  GFOV.  The  first  viewpoint,  placed  at  the 
top  of  the  display  panel,  was  a  fully  egocentric  “immersed”  perspective  similar  to  what  would  be 
seen  by  a  pilot  looking  out  of  the  windscreen  (Wickens  &  Prevett,  1995).  The  second  viewpoint, 
placed  at  the  bottom  of  the  display  screen,  was  presented  from  an  exocentric  perspective  similar 
to  that  of  the  3D  display  described  above.  However,  in  order  to  provide  a  more  global  view  of  the 
world,  the  viewpoint  distance  with  the  exocentric  display  was  increased  from  a  value  of  40,000 
feet  to  45,000  feet  and  the  elevation  angle  was  increased  from  30°  to  45°.  Also,  because 
participants  would  presumably  use  the  egocentric  viewpoint  for  primary  flight  control,  the 
azimuth  offset  in  this  display  was  decreased  from  fifteen  to  zero0. 
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Figure  2.  2D  coplanar  display. 
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Figure  3.  3D  exocentric  display.  The  next  waypoint  is  shown  by  the  cube  mounted  on  the  pole 
ahead  of  the  aircraft. 


8 


Figure  4.  Projection  parameters  of  the  exocentric  viewpoint. 
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Figure  5.  3D  split  screen  display.  The  upper  panel  is  the  immersed  egocentric  view. 
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To  aid  the  participants  in  their  navigation  task  there  were  three  additional  augmentations 
to  the  standard  displays  described  above:  1)  An  Attitude  Directional  Indicator  (ADI)  was  placed 
at  the  top  of  the  each  display  to  aid  inner-loop  flight  control.  2)  A  digital  readout  of  the  aircraft's 
current  airspeed  was  also  placed  to  the  left  of  the  ADI.  3)  To  help  participants  quickly  orient  to 
the  next  waypoint,  a  series  of  concentric  rings  (set  at  one  mile  apart)  and  lines  emanating  from 
the  aircraft  was  created  which  formed  a  “dart  board”  that  surrounded  the  aircraft. 

Task 


For  each  display  condition,  all  participants  were  asked  to  navigate  as  quickly  and  directly 
as  possible  to  a  specific  waypoint  which  was  depicted  in  the  environment  by  a  flashing  yellow 
cube.  To  aid  altitude  encoding,  a  reference  line,  extending  from  the  bottom  of  the  cube  to  the  grid 
below,  was  added  to  each  waypoint  (Ellis,  McGreevy,  &  Hitchcock,  1987).  For  each  navigation 
leg  only  one  waypoint  appeared  in  3D  space.  As  a  result,  once  the  participant  successfully 
intercepted  the  specified  waypoint  it  vanished  and  a  second  waypoint  appeared.  Participants  were 
asked  to  navigate  to  a  total  of  eight  waypoints  for  each  display  format.  All  waypoints  were 
separated  by  a  distance  of  24,000  feet  and  all  were  positioned  randomly  in  both  location  and 
altitude.  Once  the  8th  waypoint  was  intercepted  the  session  would  terminate.  Hence,  the  only 
break  that  occurred  was  between  display  conditions. 

These  eight  legs  within  each  flight  were  composed  of  the  following: 

1)  Two  straight  legs.  For  these  legs  no  excessive  lateral  or  vertical  maneuvers  were 
required  to  reach  the  next  waypoint.  For  example,  for  a  given  straight  leg  participants  needed 
only  to  descend  or  climb  to  reach  the  next  waypoint.  That  is,  no  hazard  volumes  or  terrain  were 
present  in  the  participant’s  path. 

2)  Six  maneuvering  legs  due  to  permanent  hazards.  In  order  to  reach  the  next  waypoint  as 
quickly  as  possible  participants  needed  to  circumnavigate  permanent  hazards  which  were  present 
in  their  path.  This  would  necessitate  a  decision  on  the  participant’s  part  as  to  whether  a  climb, 
descent,  turn,  or  combination  climbing  tum/descending  turn  would  result  in  the  shortest 
navigation  time  to  the  next  waypoint.  Of  the  six  maneuvering  legs,  two  could  be  navigated  most 
rapidly  with  a  lateral  maneuver,  two  with  a  vertical  maneuver,  and  two  required  a  combination  of 
lateral  and  vertical  maneuvers. 

In  addition,  within  the  six  maneuvering  legs  there  were  two  legs  in  which  an  external 
threat  appeared  at  some  random  position  that  was  in  the  view  for  all  three  displays,  and  two 
within  which  a  hazard  volume  appeared  in  the  participant’s  path.  The  appearance  of  a  hazard 
volume  required  some  added  maneuvering  in  addition  to  that  which  was  required  due  to  the 
presence  of  a  permanent  hazard. 

Performance  Measures 


1)  Total  time  for  each  leg:  The  timing  for  the  individual  legs  began  when  the  next 
waypoint  began  flashing  and  ended  when  the  participants  intercepted  the  specified  waypoint. 
Participants  were  encouraged  to  fly  the  shortest  route  possible  to  each  waypoint.  Hence,  since 
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airspeed  was  constant,  excessive  time  measures  would  indicate  a  deficiency  in  local  guidance,  as 
well  as  difficulties  in  efficiently  circumnavigating  the  hazards. 

2)  Contact  with  surrounding  terrain/hazards:  The  total  number  of  contacts  was 
measured  within  each  leg  and  the  total  time  within  the  terrain/hazard  volume  was  also  measured 
for  each  leg.  If  contact  did  occur  participants  were  immediately  alerted  with  a  beeping  tone.  For 
each  particular  display  condition,  this  measure  was  designed  to  assess  participants’  awareness  of 
surrounding  hazards. 

3)  External  threat  identification:  At  unexpected  times  during  the  navigation  task,  an 
external  threat  appeared  on  the  participant’s  display  in  the  form  of  a  red  aircraft  icon.  Participants 
were  asked  to  verbally  identify  the  location  of  the  external  threat  as  quickly  as  possible  according 
to  the  following  format: 

A)  O’clock  position:  1  to  12. 

B)  Altitude: 

HIGH:  meaning  the  threat  is  higher  than  current  altitude. 

LEVEL:  the  threat  is  at  current  altitude. 

LOW:  the  threat  is  below  current  altitude. 

C)  Trend: 

CLOSING:  the  threat  is  approaching. 

OPENING:  the  threat  is  receding. 

The  experimenter  encoded  the  verbal  responses  onto  the  computer.  Both  the  latency  of  the  first 
word  of  the  vocal  utterance  and  the  accuracy  of  response  was  measured  for  each  individual 
response.  Accuracy  of  response  was  collected  by  comparing  the  participant’s  actual  response  to 
the  ideal  response.  The  ideal  response  was  based  on  the  pilot’s  XYZ  position  and  trend  at  the 
moment  the  threat  appeared. 

4)  Pop-up  conflicts:  As  with  the  external  threats,  at  unexpected  times  during  the 
navigation  task,  a  hazard  volume  would  appear  in  the  participant’s  path.  The  participant’s  task 
was  to  decide  on  a  maneuver  that  would  circumnavigate  the  hazard  as  quickly  as  possible.  When 
an  appropriate  maneuver  was  decided  upon,  participants  keyed  a  button  on  the  joystick  and 
immediately  commenced  the  maneuver.  A  time  measure  was  recorded  in  seconds  for  both  length 
of  decision  time  and  length  of  maneuvering  time  around  the  hazard.  Both  measures  began  at  the 
moment  the  hazard  volume  appeared.  There  were  two  pop-up  conflicts  that  appeared  for  each 
display  format.  In  order  to  circumnavigate  the  pop-up  hazard  as  quickly  as  possible,  one  required 
a  lateral  maneuver  while  the  other  required  a  vertical  maneuver. 

5)  Flight  path  position:  The  participant’s  XYZ  position  was  recorded  every  five  seconds. 
This  was  done  in  order  to  later  determine  the  quality  of  maneuvers  (i.e.,  climb,  turn)  that  were 
used  to  navigate  around  the  hazards  presented. 
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Design  and  Procedure 


A  repeated  measures  design  was  used  so  all  30  participants  were  randomly  broken  into 
six  groups,  each  of  which  were  randomly  assigned  to  one  of  six  display  condition  orders.  These 
were  defined  by  cycling  twice  through  the  three  display  conditions,  each  time  in  a  different  order. 
Upon  arrival,  participants  were  given  instructions  explaining  the  purpose  of  the  experiment  and 
the  tasks  they  would  encounter.  After  reading  the  instructions,  participants  were  seated  in  a  dimly 
lit  room  in  front  of  the  IRIS  workstation. 

Three  practice  sessions  and  three  experimental  sessions  were  conducted  in  a  one-day 
experiment.  During  each  of  the  practice  sessions  participants  were  encouraged  to  fly  the  shortest 
route  possible  to  each  waypoint.  The  experimenter  ensured  that  participants  understood  the 
different  features  shown  with  each  particular  display  format.  After  completing  the  practice 
session  the  experimenter  left  the  room  and  the  experimental  session  began.  During  each  of  the 
experimental  sessions,  participants  experienced  two  external  threat  identification  tasks  and  two 
pop-up  conflict  tasks.  The  location  of  the  external  threats  and  hazard  volumes  was  altered  for 
each  experimental  session  with  a  new  display  format. 

Results:  Experiment  1 

The  data  for  the  baseline  experiment  are  shown  in  Figures  6-8  in  the  white  bar  graphs.  (The 
dark  bar  graphs  represent  the  data  for  Experiment  2,  and  will  be  described  subsequently.)  Shown  in 
Figure  6  is  the  mean  time  to  traverse  all  waypoints  within  a  mission  as  a  function  of  display  type. 
The  significant  effect  of  display  (F(2,40)  =  20.7,  p  <  0.01)  revealed  the  clear  (and  predicted) 
advantage  of  the  immersed,  split  screen  display  for  this  local  guidance  task,  relative  to  the  costs 
shown  by  the  coplanar  format  (reflecting  the  lateral  and  vertical  cross  panel  integration  of  costs), 
and  the  costs  shown  by  the  3D  exocentric  format  (reflecting  ambiguity  in  estimating  ownship 
position  and  heading).  Subsequent  analysis  revealed  that  the  major  deficiency  of  the  exocentric 
format  was  in  the  fine  maneuvering  required  to  capture  the  3D  spatial  envelope  surrounding  each 
waypoint  (see  also  Andre,  Wickens,  Moorman,  &  Boschelli,  1991). 

Further  analyses  revealed  differential  costs  for  the  coplanar  and  exocentric  displays  on 
vertical  (climbing  or  descending)  versus  lateral  (turning)  legs.  Performance  with  the  coplanar 
display  suffered  the  greatest  costs  on  legs  involving  only  vertical  maneuvers  (F(l,20)  =  10.35;  p^< 
0.01)  while  the  exocentric  display  showed  the  greatest  costs  on  lateral  maneuvering  legs  whether 
these  were  level  (F(l,20)  =  31.9,  p^<  0.01)  or  also  involved  vertical  maneuvering  )  F  =  13.93,j>  < 
0.01). 


Figure  7  presents  the  amount  of  time  spent  within  a  hazard  volume  (e.g.,  ownship  has 
temporarily  penetrated  a  radar  coverage  zone).  While  these  times  are  relatively  short  (around  1 
second  in  a  several  minute  flight.  They  are  still  non-zero.  Again  they  reflect  the  marginally 
significant  advantage  of  the  split  screen  display,  and  its  ego-referenced  guidance  panel  in 
supporting  this  aspect  of  navigation  (F=  2.93;  p_<  0.01).  This  cost  was  greatest  for  the  coplanar 
display,  and  on  those  legs  requiring  lateral  maneuvers. 
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Figure  6.  Mean  travel  time/leg.  White  bar  graphs  represent  data  from  Experiment  1.  Black  bar 
graphs  are  data  from  Experiment  2. 


Figure  7.  Mean  time  contacting  hazard  volumes  per  leg. 
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Mean  Response  Time  to  Pop-Up  Threat  vs  Display 


Figure  8.  Mean  response  time  to  aircraft  threat. 


Figure  8  presents  the  time  required  for  pilots  to  respond  to  a  traffic  threat,  as  inferred  by  the 
time  to  initiate  the  first  component  of  their  identification  response.  In  this  case  the  significant  effect 
(F=  19.85;  p_<  0.01)  reverses  the  pattern  shown  in  the  guidance  measures  plotted  in  Figures  6  and  7, 
and  indicates  a  substantial  cost  imposed  by  the  split  screen  display.  A  subsequent  debriefing  of  the 
participants  revealed  that  the  cost  was  attributable  to  the  “compelling”  nature  of  the  immersed 
screen  (on  top),  which  appeared  to  inhibit  participants  from  allocating  much  of  their  visual  attention 
to  monitoring  the  global  hazard  awareness  screen  below.  Note  in  contrast  that  on  the  coplanar 
display,  information  necessary  for  the  primary  guidance  task  was  distributed  across  both  panels,  and 
on  the  exocentric  display  there  was  only  one,  unified  panel.  A  comparison  between  the  coplanar 
and  exocentric  format  revealed  the  latter,  with  its  single  integrated  panel,  to  have  a  marginally 
significant  benefit  for  response  time  (F(l,19)  =  3.66,  p  <  0.08). 

Our  assessments  of  the  accuracy  of  estimating  azimuth  and  closure  of  the  intruder  traffic 
revealed  no  differences  between  the  three  display  formats.  However,  Figure  9  suggests  a  marked 
influence  on  the  estimation  of  intruder  altitude  (F=  7.54;  p_<  0.01).  The  coplanar  display,  with  its 
linear  depiction  of  altitude  supported  best  performance.  We  infer  that  both  3D  displays  suffered  the 
cost  of  ambiguity  in  estimating  the  vertical  dimension.  We  also  inferred  that  the  split  screen  cost 
was  greater  than  the  exocentric  cost,  because  the  dimensions  of  the  perspective  display  at  the 
bottom  of  the  split  screen  upon  which  all  targets  were  viewable,  were  considerably  smaller  than 
those  of  the  exocentric  display,  with  a  resulting  loss  in  resolution.  This  resolution  reduction  resulted 
because  the  global  3D  view  in  the  split  screen  display  had  to  share  real  estate  with  the  immersed 
view. 
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Screen 

Mean  Pop-Up  Threat  Altitude  Judgment  Error  vs  Display 


Figure  9.  Mean  error  in  altitude  judgment  of  aircraft  threat. 


Discussion:  Experiment  1 

The  results  of  Experiment  1  revealed  some  of  the  predictable  shortcomings  of  each 
prototype  display  format:  the  coplanar  and  exocentric  display  supported  less  efficient  guidance  and 
navigation,  although  each,  we  assume,  for  different  reasons  (vertical  and  lateral  integration 
demands  for  the  coplanar  display,  and  ambiguity  in  estimating  ownship  for  the  exocentric  display). 
Replicating  earlier  findings  of  Merwin  et  al.  (1997),  it  is  interesting  to  note  that  the  cost  of  the 
exocentric  display  was  much  greater  on  legs  that  required  combined  vertical  and  lateral 
maneuvering,  than  on  straight  and  level  legs,  in  which  constraints  could  be  applied.  It  is  not  entirely 
clear  why  the  cost  of  scanning  and  integration  for  the  coplanar  display  was  greatest  on  vertical 
maneuvering  legs. 

With  regard  to  hazard  awareness,  both  of  the  3D  displays  suffered  ambiguity  in  estimating 
the  altitude  of  traffic.  Furthermore,  hazard  awareness  with  the  immersed  display  suffered  from 
inappropriate  allocation  of  visual  attention,  as  we  assume  that  visual  scanning  to  the  hazard  display 
was  drawn  away  to  the  more  “compelling”  and  information-rich  source  of  navigational  guidance. 
Efforts  to  address  these  various  display  deficiencies  were  undertaken  in  Experiment  2. 

Experiment  2:  Cognitive  Engineering  Enhancements 

Cognitive  engineering  is  a  discipline  whose  goals,  in  part,  are  directed  toward  engineering 
human-system  performance  improvements,  based  upon  a  clear  understanding  of  the  cognitive 
strengths  and  limitations  of  the  human  user  (Rasmussen,  Pejtersen,  &  Goodstein,  1995).  In  the 
current  application,  we  applied  cognitive  task  analysis  to  pilot-display  interaction  in  Experiment  1. 
This  analysis  was  based  upon  participant  debriefing,  performance  records,  and  our  own  knowledge 
of  cognitive  engineering.  We  identified  these  limitations  in  each  display  suite.  We  then  developed 
enhancements  based  upon  different  aspects  of  attention  theory  and  the  concept  of  visual  momentum 
(Woods,  1984),  and  applied  these  selectively  to  each  display  format,  to  be  evaluated  in  its  enhanced 
form  in  Experiment  2.  We  describe  the  enhancement  to  each  format  as  follows. 
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Split  screen  display.  The  deficiencies  of  the  split  screen  display  in  supporting  traffic 
awareness  resulted  from  the  inappropriate  allocation  of  attention  to  the  immersed  display  panel 
(Kahneman,  1973;  Wickens,  1989).  To  address  this  limitation  in  Experiment  2,  we  provided  an 
auditory  annunciation  for  the  appearance  of  each  target  on  the  radar  display.  This  feature  was 
incorporated  with  all  three  displays,  but  we  reasoned  that  its  benefits  would  be  greatest  for  the  split 
screen. 


Coplanar  display.  The  expected  difficulties  of  navigating  with  this  display  were  attributed  to 
difficulties  in  integrating  the  lateral  maneuvering  aspects  with  the  vertical.  Hence,  two  corrective 
techniques  were  applied  in  Experiment  2,  both  based  upon  the  concept  of  visual  momentum 
(Woods,  1984;  Aretz,  1991),  designed  to  provide  pilots  with  a  better  understanding  of  how 
overlapping  entities  depicted  in  the  top  (lateral)  panel  were  depicted  in  the  bottom  (vertical  profile) 
panel.  First,  the  vertical  depiction  was  rotated  90°,  to  present  a  view  from  behind,  rather  than  from 
the  side.  This  orientation  has  the  advantage  of  providing  vertical  alignment  or  correspondence 
between  different  representations  of  the  same  geographical  features  in  the  two  displays.  That  is,  the 
“God’s  eye”  representation  of  a  given  entity  in  the  upper  display  will  be  directly  above  its 
corresponding  profile  representation  in  the  bottom  display.  This  alignment  was  expected  to  help  to 
cognitively  “link”  the  two  panels.  Second,  greater  heterogeneity  of  color  coding  of  the  features  was 
employed.  By  therefore  making  each  feature  relatively  more  distinct  and  unique  (in  relation  to  its 
neighbors),  such  coding  enables  the  pilot  to  gain  greater  certainty  as  to  the  match  between  the  two 
panels. 


Exocentric  display.  The  deficiencies  of  this  display  were  those  that  had  been  observed 
previously,  and  concerned  ambiguities  in  estimating  the  position  of  ownship  when  nearing  a 
waypoint,  and  when  navigating  close  to  the  3D  hazards.  To  address  these  limitations,  command 
vertical  guidance  information  was  presented  at  each  waypoint,  in  the  form  of  an  arrow  that  pointed 
up  or  down,  depending  on  the  required  vertical  correction.  Capitalizing  on  the  cognitive  benefits 
offered  by  3D  prediction  (Wickens,  Haskell,  &  Harte,  1989),  ownship  was  also  provided  with  an 
inverted  triangle  predictor  symbol.  This  predictor,  extending  approximately  10  seconds  ahead  with 
its  apex  attached  to  ownship's  nose,  was  designed  to  change  coloration  any  time  it  penetrated  a 
volume,  thereby  providing  3D  spatial  prediction  of  clearances.  These  enhancements  are  shown  in 
Figure  10.  Finally,  because  of  the  ambiguity  with  which  altitude  information  of  the  traffic  threats 
was  encoded  in  Experiment  1,  in  the  current  experiment,  intruder  altitude  was  provided  with  color 
coding:  red  if  below  ownship,  black  if  level,  and  white  if  above. 

Design  and  Procedures 

In  contrast  to  Experiment  1,  a  between-subjects  design  was  employed  for  Experiment  2. 
Nine  pilots  were  assigned  to  each  of  the  three  display  conditions.  Each  pilot  then  flew  a  total  of 
three  missions,  with  each  mission,  as  in  Experiment  1,  consisting  of  a  series  of  waypoints,  hazards, 
and  traffic  threats.  All  other  procedures  were  identical  to  those  employed  in  Experiment  1. 
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Figure  10.  The  “Wedge”  predictor. 


Results:  Experiment  2 

The  time  required  to  navigate  a  mission  (and  hence,  also  a  measure  of  the  efficiency  of 
flight  path)  is  shown  by  the  dark  bars  in  Figure  6.  Two  features  are  evident  from  these  data.  First, 
across  all  displays,  flight  time  was  substantially  shorter  than  it  was  in  Experiment  1.  Second,  the 
greatest  reductions  in  time  were  realized  by  the  two  displays  that  had  been  “deficient”  in 
Experiment  1,  the  coplanar  and  exocentric  display.  Hence,  the  consequence  of  these  improvements 
is  that  there  was  no  difference  across  displays  in  navigation  time  observed  in  Experiment  2  (F  <  1). 
We  assume  that  the  predictive  tools  assisted  with  disambiguating  the  navigational  paths  in  the 
exocentric  display,  and  that  the  visual  momentum  links  between  the  two  panels  of  the  coplanar 
display  facilitated  the  pilots’  navigational  awareness  in  estimating  3D  trajectories. 

Figure  7  depicts  the  total  amount  of  time  spent  flying  within  a  hazard  volume.  Here  again, 
the  display  augmentations  served  to  improve  performance,  once  again  achieving  greater 
equivalence  of  the  performance  across  the  three  displays,  by  showing  the  greatest  improvement  on 
the  coplanar  display  which  had  supported  the  poorest  performance  in  Experiment  1.  Here  once 
again,  the  visual  momentum  augmentation  proved  quite  helpful.  The  Experiment  2  data  revealed  a 
significant  effect  of  display  on  contact  time  (F(2,24)  =  6.46;  p  <  0.01)  with  the  exocentric  display 
now  showing  the  greatest  cost. 

Figure  8  depicts  the  response  time  to  detect  (and  begin  identifying)  the  pop-up  threat.  As 
with  the  previous  two  measures,  an  overall  performance  improvement  was  observed,  and  this 
improvement  was  greatest  for  the  split  screen  display  for  which  the  attention  allocation  deficiencies 
had  been  greatest  in  Experiment  1 .  The  data  for  Experiment  2  revealed  a  still  significant  effect  of 
display  format  (F(2,159)  =  8.16;  p_<  0.01)  still  favoring  the  exocentric  display. 

Figure  9  shows  the  error  in  altitude  judgment  across  the  three  displays,  and  reveals  the 
manner  in  which  the  color  coding  of  altitude  almost  entirely  eliminated  any  errors  in  this  task.  In  a 
sense,  this  improvement  is  easily  predictable  since  it  would  have  been  possible  for  pilots  to  make 
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the  required  altitude  judgments  strictly  by  attending  to  the  symbolic  color  coding  of  the  threat 
symbol,  and  ignoring  vertical  position  altogether. 


Discussion:  Experiment  2 

It  is  clear  that  the  enhancements  applied  to  the  displays,  based  upon  cognitive  task  analyses 
of  display  deficiencies  in  Experiment  1,  were  successful.  In  all  cases  performance  on  the  various 
tasks  improved  or  remained  constant,  without  encountering  compensatory  costs  in  other 
performance  aspects.  One  caution,  however,  should  be  exercised  in  interpreting  these  results. 
Because  of  the  limited  availability  of  participants,  some  of  the  pilots  had  participated  in  Experiment 

1,  and  for  these  individuals,  the  differences  between  displays  in  Experiment  2  was  confounded  with 
practice.  However,  this  was  only  a  small  percentage  of  our  participant  population,  and  they  were 
equally  balanced  across  the  three  display  conditions  in  Experiment  2.  Furthermore,  a  subsequent 
analysis  revealed  that  the  benefits  that  these  “repeating”  participants  realized  were  not  substantially 
greater  than  the  benefits  of  the  non-repeaters. 

The  most  important  point  to  note,  in  any  case,  is  that  generally  the  “fixes”  offered  to  the 
displays  served  to  equalize  performance  across  them,  thus  alleviating  the  problem  that  often  occurs 
in  engineering  design;  namely  that  a  “fix”  applied  to  address  one  aspect  of  performance,  succeeds 
only  in  creating  another  problem  elsewhere.  We  believe  that  a  careful  application  of  engineering 
principles  that  address  the  perceptual  cognitive  mechanisms  underlying  performance,  rather  than 
just  addressing  performance  deficiencies  itself,  can  alleviate  some  of  the  tradeoffs  that  are  often 
observed.  Cognitive  engineering  can  accomplish  this  as  we  believe  we  have  demonstrated  here. 

Finally,  reviewing  the  overall  effects  observed  across  all  dependent  variables  in  Experiment 

2,  the  results  point  to  the  split  screen  display  as  the  most  effective.  A  split  screen  design  such  as  that 
used  here  would  seem  to  offer  advantages  in  terms  of  both  spatial  compatibility  and  the  pilot’s 
cognitive  needs  as  discussed  as  follows. 

First,  presentation  of  the  immersed  panel  at  the  top  provides  it  with  a  compatible  location 
with  the  forward  field  of  view  as  the  pilot  moves  through  the  airspace  (and  indeed  in  close 
proximity  to  a  similarly  formatted  conformal  HUD).  Presentation  of  the  more  global  panel  below  is 
more  compatible  with  the  view  a  pilot  would  get  looking  downward  to  visually  capture  a  wider 
region  of  space.  Hence,  this  display  location  is  compatible  with  the  location  of  information  relative 
to  the  pilot  in  real  space. 

Second,  the  two  views  of  the  split  screen  display  can  be  understood  to  provide  a  cognitive 
match  to  the  way  humans  perceive  and  understand  the  environment.  Perception  has  often  been 
described  as  having  both  “global”  and  “local”  aspects  (Neisser,  1976;  Navon,  1987).  At  the  same 
time,  cognitive  engineering  theory  has  stressed  the  need  for  human  operators  (here  pilots)  to  interact 
with  their  work  domain  at  both  higher  and  lower  levels  of  abstraction  (Rasmussen  et  al.,  1995; 
Vicente  &  Rasmussen,  1990).  The  global  and  local  panels  of  the  split  screen  display  map  nicely  to 
the  higher  and  lower  levels  (respectively)  of  this  abstraction  hierarchy. 
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Experiment  3 


We  summarize  briefly  below  additional  details  of  a  third  experiment  performed  under  the 
contract.  The  three  baseline  displays  employed  in  Experiment  1  were  again  altered,  according  to 
cognitive  engineering  principles.  (1)  Visual  flashing  of  targets  (rather  than  auditory  cueing  as  in 
Experiment  2),  (2)  Were  employed  in  an  effort  to  facilitate  intruder  detection  particularly  in  the 
split  screen  (SS)  display.  The  elevation  angle  of  the  bottom  (global)  display  of  the  split  screen 
display  was  raised  to  90°,  (3)  Object  vectors  were  incorporated  on  the  aircraft  representations  in  the 
exocentric  display  to  facilitate  the  judgments  of  relative  azimuth,  elevation,  and  range  (Figure  1 1). 
Based  upon  previous  research  on  object  displays  and  their  emergent  features  (Wickens  &  Carswell, 
1995;  Carswell  &  Wickens,  1996),  it  was  reassured  that  these  judgments  of  spatial  awareness  could 
be  made  perceptually,  on  the  basis  of  the  configuration  of  the  geometric  object  formed  by  ownship, 
intruder,  and  connecting  the  vectors.  Full  details  of  the  rationale  and  design  can  be  found  in 
Davenport  and  Wickens  (1997). 

The  full  results  of  Experiment  3  are  presented  in  Davenport  and  Wickens  (1997).  The 
general  conclusion  reached  regarding  the  overall  performance  of  the  displays  in  experiments  is  that 
the  3D  exocentric  display,  even  after  its  enhancements,  did  not  support  either  local  or  global  tasks 
as  well  as  either  of  the  other  two  display  formats.  The  ability  of  the  object  display  enhancements  to 
support  improved  awareness  of  intruder  location  was  partially  successful.  Altitude  judgments  were 
improved,  but  bearing  judgments  were  not.  The  limited  success  may  have  been  attributable  to  the 
use  of  perspective  rather  than  parallel  geometry  in  the  display  (see  Figure  11). 


Figure  1  la.  Evolution  of  object  display  elements  in  a  3D  exocentric  display,  (a)  Judging  the  relative 
altitude  of  a  target  to  ownship  (o)  using  only  a  vertical  height  (altitude  vector)  is  difficult  because  of 
the  display’s  inherent  ambiguity,  (b)  Illustrates  the  effect  of  adding  a  horizontal  air  vector  between 
ownship  and  the  target.  The  relative  altitude  is  still  not  apparent,  (c)  Shows  the  effect  of  making  an 
object  by  using  a  horizontal  ground  vector  to  close  the  figure.  The  resultant  shape  should  make  the 
judgment  of  relative  altitude  more  accurate  and  timely  than  unenhanced  3D  exocentric  displays. 
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Figure  lib.  Exocentric  display  with  different  examples  of  intruders  and  object  display  elements. 


There  was  a  benefit-cost  tradeoff  between  the  remaining  two  formats  (coplanar  and  split 
screen).  The  SS  display  suite  shows  a  definite  benefit  for  local  guidance  tasks  which  is  balanced  by 
relatively  poor  global  awareness  support  for  event  detection  and  vertical  axis  judgments  compared 
to  the  2D  coplanar  suite.  Therefore,  a  conclusion  regarding  which  of  these  two  display  formats 
provides  the  best  overall  support  for  both  local  and  global  tasks  must  take  into  account  the  tasks 
performed  and  frequency  with  which  they  are  performed.  The  local  awareness  and  guidance  tasks 
are  continuous  in  terms  of  mission  demands  whereas  global  SA  tasks  are  most  likely  to  be 
intermittent.  We  conclude  that  the  SS  display  suite  is  most  likely  the  best  candidate  for  use,  given 
our  experimental  paradigm,  since  it  provided  the  best  support  for  the  continuous  task  of  local 
awareness  and  guidance  and  poor  support  for  some,  but  not  all,  of  the  global  SA  tasks.  We  believe, 
however,  that  the  split  screen  format  evaluated  in  Experiments  1  and  2,  with  the  perspective  global 
view,  is  probably  superior.  Finally,  we  recognize  that  the  flashing  (visual)  cues  of  the  intruder 
aircraft  were  less  salient  than  the  auditory  cues  used  in  Experiment  2,  and  hence  attention  allocation 
to  the  upper  (immensed)  panel  of  the  split  screen  display  remains  a  problem  to  be  addressed. 
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